- H\(_0\): No significant association between Income Level and Subscription Status
- H\(_1\): There is a significant association between Income Level and Subscription Status
As datasets grow in complexity, we often face the Curse of Dimensionality — a phenomenon where high-dimensional data becomes sparse, making distance-based algorithms like KNN perform poorly. The performance of machine learning models degrades as the number of features increases without a proportional increase in data.
Key Question: How can we reduce the number of features while preserving the predictive power of our model?
Two main approaches combat this curse:
In this lecture, we focus on Filter Methods — fast, model-agnostic statistical techniques for selecting the most informative features before training any machine learning model.
Given a feature matrix \(X \in \mathbb{R}^{n \times d}\) with \(n\) samples and \(d\) features, and target variable \(y \in \mathbb{R}^n\), our goal is to select a subset \(S \subseteq \{1, 2, ..., d\}\) such that features in \(S\) are most predictive of \(y\).
Exponential Search Space: There are \(2^d\) possible subsets of \(d\) features. For \(d = 30\), we must consider \(2^{30} = 1,073,741,824\) subsets — exhaustive search is computationally infeasible!
| Characteristic | Filter Methods | Wrapper Methods |
|---|---|---|
| Timing | Preprocessing step, independent of ML algorithm | "Wraps" around a specific ML algorithm |
| Evaluation | Statistical measures (correlation, chi-square) | Actual model performance (cross-validation) |
| Speed | Fast and scalable | Slower but more accurate |
| Model Dependence | Algorithm-agnostic | Tailored to specific algorithm |
Note: A third category called "Embedded Methods" (e.g., LASSO, Random Forest feature importance) exists where feature selection happens during model training.
The simplest filter method removes features with very low variance:
Important: Standardization or normalization is required before applying variance threshold so that the same threshold works for all features. A threshold between 0.01 and 0.1 is generally effective.
The Chi-Square test measures the association between a categorical feature and a categorical target variable.
Where \(c\) is the degree of freedom, \(O_i\) is the observed frequency, and \(E_i\) is the expected frequency in cell \(i\).
The expected frequency is computed as:
ANOVA (Analysis of Variance) is used when:
Where \(K\) = number of classes, \(N\) = total samples.
Higher F-statistic indicates greater difference between class means, making the feature more informative.
Mutual Information measures how much knowing one variable reduces uncertainty about another. It works with any feature type (categorical or continuous).
| Method | Target Type | Feature Type | Captures Non-linear | Model-agnostic |
|---|---|---|---|---|
| Variance Threshold | Any | Any | No | Yes |
| Correlation | Continuous | Continuous | No | Yes |
| Chi-Square | Categorical | Categorical | No | Yes |
| Mutual Information | Any | Any | Yes | Yes |
| ANOVA F-test | Categorical | Continuous | No | Yes |
Problem: For \(d = 3, 4,\) and \(5\) features, how many possible subsets exist? List all subsets for \(d = 3\).
Solution:
For \(d = 3\) with features \(\{A, B, C\}\):
Dataset: Examining the relationship between "Income Level" (Low, Medium, High) and "Subscription Status" (Subscribed, Not Subscribed).
| Income Level | Subscribed (O) | Not Subscribed (O) | Row Total |
|---|---|---|---|
| Low | 20 | 30 | 50 |
| Medium | 40 | 25 | 65 |
| High | 10 | 15 | 25 |
| Column Total | 70 | 70 | 140 |
Expected Values:
Using the contingency table from Example 2, compute the Chi-Square statistic and determine if Income Level is a significant predictor of Subscription Status at \(\alpha = 0.05\).
| Income Level | Subscribed (E) | Not Subscribed (E) |
|---|---|---|
| Low | 25 | 25 |
| Medium | 32.5 | 32.5 |
| High | 12.5 | 12.5 |
Formula: \(E_{ij} = \frac{\text{Row}_i \times \text{Column}_j}{\text{Grand Total}}\)
Total: \(\chi^2 = 6.462\)
Critical value at \(\alpha = 0.05\), \(df = 2\): 5.991
Since \(6.462 > 5.991\), we reject H\(_0\).
Conclusion: There is a significant association between Income Level and Subscription Status. This feature would be selected by the Chi-Square filter method. Higher \(\chi^2\) values indicate a stronger feature-target relationship.
Problem 1: Variance Threshold Decision
You have 4 features with the following variances after standardization: A=0.15, B=0.003, C=0.08, D=0.001. If you apply VarianceThreshold with threshold=0.01, which features will be selected?
Features with variance >= 0.01 are selected:
Selected features: A and C
Problem 2: ANOVA F-test Interpretation
You compute ANOVA F-statistics for three features predicting a binary target:
Using \(\alpha = 0.05\), which features are significant? Rank them by importance.
Compare each p-value to \(\alpha = 0.05\):
Ranking by F-statistic (higher = more important):
Problem 3: Method Selection
For each scenario, identify the most appropriate filter method:
Problem 4: Mutual Information Comparison
Suppose you calculate Mutual Information scores for four features:
If you need to select the top 2 features, which do you choose? What does MI = 0.0 tell you about Feature A?
Top 2 features by MI score: Feature D (0.78) and Feature B (0.45)
Feature A with MI = 0.0: This indicates that Feature A is completely independent of the target variable. Knowing Feature A provides zero information about the target. It should be dropped.
Test your understanding of Filter Methods. Select the best answer for each question.
Q1. What is the primary advantage of filter methods over wrapper methods?
Q2. How many possible feature subsets exist for a dataset with 20 features?
Q3. Which filter method is appropriate for a categorical feature and categorical target?
Q4. What does a higher F-statistic in ANOVA indicate about a feature?
Q5. Which filter method can handle both categorical and continuous features?